Outside attacks that attempt to compromise a computer system are an increasingly common and important threat. Computer programs and applications on such compromised systems can generally only be used to cause real damage by exploiting system calls, making the system call interface the ideal point to detect and control various types of attacks. Consequently, system call monitoring has been a widely used technique for detecting and quarantining compromised applications, in an effort to minimize any damage that could be caused.
Prior system call monitoring approaches have been based on developing a model, or policy, of an application's normal system call behavior, and then halting execution when an application deviates from its modeled behavior. Policy checking and enforcement are security-critical. Hence, in prior systems, such functions are performed entirely within the operating system kernel, or within the operating system kernel and in conjunction with a separate, protected policy server (daemon). Both such approaches require large-scale changes to the kernel. In addition, the former can have unacceptably high execution costs, while the latter can result in a substantially more complex kernel, which then has further associated increases in execution overhead.
Accordingly, there is a need for a system and method for enforcing application security policies that addresses certain problems of existing technologies.